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Research  and  study  in  producing  this  bibliograpln.c  paper  were 
supported  by  a  grant  from  the  Graham  Foundation  for  Advanced 
Studies  in  the  Fine  Arts,   for  Xirork  tot-rards  a  Ph.D.  in  Architecture 
at  the  University  of  California  at  Berkeley.     Mnor  emphasis 
work  in  Information  System  Design  is  the  basis  for  this  survey 
of  tvra  approaches  to  information  retrieval. 


In  Ms  characterization  of  information  systems  for  planning 
and  design,  Horst  Rittel  distinguishes  data  from  information. 
Data  is  in  the  form  of  documents|  information  is  a  process. 
One  way  of  vievdng  the  distinction  is  to  say  that  information 
results  from  the  inteipretation  of  data  vathin  one's  worldview. 
For  a  process  of  information  to  have  taken  place,  there  must 
have  been  a  change  in  knowledge:     contradiction,   corroboration, 
alteration,   displacement!  but  a  change. 

In  the  field  of  information  system  design,   a  distinction  is  draxm 
between  question  answering  systems  and  literature  searcliing 
systems.     One  question  to  be  dealt  with  in  the  design  of  a 
planning  and  design  information  system  is  the  choice  between 
these  two  morphologies 5   there  are,  of  course,  many  additional 
issues  to  be  resolved  that  go  beyond  this  distinction. 

In  tliis  bibliographic  paper,  the  two  modes  of  information  retrieval, 
the  question  answering  system  and  the  literature  searcliing 
s^J-stem,   are  defined,   and  the  literature  dealing  xd.th  each  is 
surveyed.     It  is  suggested  that  the  most  useful  approach  for  use 
in         planning  and  design  infomation  systems  is  the  literature 
searcMng  (i.e.^   reference  retrieval  systems,   document  retrieval 
systems)   approach, since  it  is  more  "friendly"  to  the  interpretation 
of  data  T'dtliin  a  user's  worldview  than  the  more  self-contained 
question-ansvjering  approach.     Two  approaches  to  information 
retrieval  are  the  question  ansTTering  system  and  the  literature 
searching  systera.     Question  ansi-iering  systems,   also  laiown 
as  fact  retrieval  systems  (Cooper,  IJoU),  information  retrieval 
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systems  (Bar-IIillel,  19^7)  and  data-providing  systeris  (Ear- 
Hillel,  i960),  provide  direct  ansiisi's  from  stored  knouledge 
to  questions;  of  the  form,    "i/hat  is  the  ansuer  to   ...?'' 
Literature  searching  systems,  also  lonoxm  as  reference- 
providing  systems   (Bar-Hillel,  I96O)  or  document  retrieval 
systems  (Cooper,  19Sk)  provide  docuroents  or  citations  in 
response  to  questions  of  the  form,    "I.Tiat  literature  pertains 
to   ...  ?"     Both  question  ansT:ering  and  literature  searclxLng 
systems  attempt  to  analyze  and  identify  the  information  that 
is  necessary  to  meeting  an  information  need,   and  onlj'-  that 
xjhich  is  necessary,   and  provide  it  to  the  user.     Information 
can  be  talcen  as  that  xj-liich  viill  change  one's  cognitive  mapi 
an  infoi-Tuation  need  exists  Tihen  one  encoimters  a  gap  in  that 
map   (iiaron,  1971-2l;2).     Rittel  defiiies  information  as  a  process 
that  changes  one's  laiowledge  (itlttel,  1971-239).     Hcuay  defines 
information  in  terms  of  determine  action  totiard  goal  states 
(McKay,  1971).     In  general,   the  question  answering  system 
atteripts  to  directly  prxivide  the  data  that  xiiill  inform,  while 
the  literature  searcliing  system  attempts  to  direct  the  user 
to  those  sources  in  which  such  data  can  be  found.     The  reference 
retrieval  or  literature  searcliing  system  does  the  first  part 
of  the  fact  retrieval  or  question  answering  system's  job. 

Of  conceivable  question  answeiung  systems,  the  simplest  are 
the  fill-in-the-blank,  look-up-device  variety,  in  wliich  the 
chief  interest  is  in  combining  the  talents  of  man  and  machine 
in  order  to  rapidly  access  specific  data  from  a  large  store. 
The  most  corrrple::  are  on  the  order  of  essay  xTriting,   conver- 
sational and  picture- describing  macliines,   almost  at  the  level 
of  Turing's  challenge  to  construct  a  tliinking  machine.     Related 
areas  are  artificial  intelligence,  heuristics  for  logical 
inference  and  automated  theorem  proving. 

Simple  and  useful  question  answering  systeros  that  deal  with 
queries  of  a  constant  form  against  a  rapic-Lj'  changing  store 
of  data  are  presently  in  use,   for  exaiiple,   the  airline  reser- 
vation system.     More  complex  and  difficult  systems  atterrpt 
to  carry  out  logical  deductions  based  on  stored  data  and 
questions  that  are  input  in  natural  language.     One  form  of 
question  answering  system  centers  around  the  concept  of  a  logical 
consequence  detector  that  stores  a  body  of  reference  sentences 
and  checks  test  sentences  against  them  in  terms  of  xAiether 
or  not  the  test  sentences  are  logical  consequences  of  the 
stoi'ed  sentences.     Cooper  (I96U)  constructed  a  question 
answering  system  based  on  the  propositions  that  sxxch  a  system 
should  accept  its  stored  sentences  end  its  test  sentences  in 
the  form  of  natural  language  and  that  it  should  possess  the 
capability  of  performing  logical  deductions  among  these 
sentences;  thus  the  key  effort  xras  toxrard  the  development 
of  a  system  of  logical  inference  among  natural  language 
sentences.     In  tliis  system,  both  test  and  stored  sentences 
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are  converted  Tx-ora  natural  languars  to  a  logical  language 
devised  for  the  system,   then  coripared.     TI  a  subset  of  the 
stored  sentences  is  found  such  that  the  test  sentence  is  a 
logical  consequence  of  the  subset,  the  system  aiistiors  "true" 
to  the  test  sentence^  if  not,   end  a  further  search  for  subsets 
jdelds  one  that  is  logically  inconsistent  Tiith  the  test 
sentence,  the  system  aiisirars  "false".     If  neither  logical 
consequence  nor  logical  inconsistency  can  be  fouiid,  the  systerr. 
answers  ''don't  Imou" .     The  bod;^'"  of  stored  sentences  iras  very 
SKall,  yet  the  nuraber  of  iaanipulations  that  was  necessary  in 
seeldng  subsets  was  vei'y  large.     Pui^tlier  progress  would  seeu 
to  rest  on  a  deeper  and  more  precise  understanding  of  logic 
and  language,   and  sor.ie  means  of  dealing  irith  the  extremely  large 
niimbers  resulting  from  the  manipulation  of  a  large  body  of 
stored  sentences.     Siiimons  (196?),  in  a  su.rvey  of  the  fii-st 
five  years  of  research  into  question  answering  systems,  concludes 
that  the  fimdaiiental  questions  in  the  area  are  those  of  hoir 
to  characterize  sentence  meaning,  hoTJ  to  deal  idth  semantic 
and/or  sjTitactic  ambiguity,   hoxj  to  make  inferences  xd-thout 
an  e:diaustive  field  coverage  of  all  combinatorial  possibilities, 
how  to  cov.ibine  partial  answers  from  tddely  separated  points 
in  the  date  body,   to  what  extent  can  or  shoxild  natural  language 
be  translated  into  formal  language  for  manipulation,   and 
xjhethcr  xXirther  work  in  question  answering  systems  should  be 
basically  theoretical  or  empirical  ba^ed  on  large  question 
answerers  built  for  test  purposes. 

The  literature  searcliing  system  accepts  qtieries  of  the  forDi, 
"irnat  literature  pertains  to   ...?",   and  outputs  bibliographies, 
stacks  of  docuraents,   or  in  some  cases  ranked  lists  of  probability 
ordered  documents.     The  enphasis  in  the  design  of  such  systems 
is  on  their  statistical  porfon.iance  over  time.     Central  topics 
are  conceptual  closeness,   relevance,   associative  search,  pro- 
babilistic inde:dng,   automated  te;rt  analysis  and  inderdng, 
measujpes  of  retrieval  effectiveness,   clustering,   and  user 
inference  searcliing. 
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Th3  functioning  of  en  information  retrieval  systen  can  be 
diagrajiimecl  as  follows: 
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The  search  models  by  xrl-dch  the  information  transfer  system 
might  proceed  can  be  characterised  as  direct  match,   siii55le 
overlap.  Boolean  overlap  and  feedback  models,     i.ssuming  a 
collection  of  documents,   each  of  Trliich  is  assigned  a  series 
of  inde:c  terms,   and  forraal  queries  composed  of  sexues  of 
index  terms: 

1.  In  the  direct  match  mode,  only  those  documents  vn.ll 
be  accessed  that  have  precisely  the  same  list  of  inde::  terms 
as  are  in  the  foi-m-al  query. 

2.  In  simple  overlap,  those  documents  1.111  be  accessed 
that  have  some  specified  minimum  number  of  coinciding  inde:: 
terms . 

3.  In  Boolean  overlap,  the  matcliing  vdll  be  based  on 
strings  of  index  terms  connected  by  "aiid",   ''or'',  and  "not'', 

so  that  accessed  docraients  ijill  have  to  meet  certain  specified 
combinations  of  possible  terms  in  the  Boolean  query. 

h.     In  the  feedback  model,  the  user  id.ll  be  informed 
of  the  nur,iber  of  docuiiients  that  uould  be  accessed  in  response 
to  his  query,   and  aliened  to  loosen  or  tighten  liis  specification 
in  order  to  increase  or  decrease  the  number  of  documents 
accessed^  and  then  to  modify  liis  query  in  response  to  irfiat  is 
accessed. 
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The  mode  of  system  response  can  furthei"  be  charscteiuzed  as 
either  dividing  the  docmnent  collection  into  tiro  parts,  one 
of  TJitLch  is  predicted  to  be  relevant  and  the  other  not;   or 
ranldns  the  entire  docuinent  collection  in  terms  of  the 
probability  of  each  docuiaent's  satisfjdng  the  information 
need. 


Search  Ho dels 


r  '  !  ---        { 
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The  problem,  of  conceptual  closeness  can  be  approached  as  being 
one  of  statistical  correlation,     Kulins  (IJolj.)  proposes  a  series 
of  measures  of  association  or  closeness  in  inde;:  space  in  terms 
of  a  statistical  measure  of  independence.     If  in  terms  of  the 
co-occurrence  of  two  index  terms,  A  and  B,  on  docioments,  the 
probability  of  the  occurrence  of  A  given  B  is  the  same  as 
the  probability  of  A,  then  A  and  B  are  said  to  be  independent. 
Kvihns  derives  a  statistical  measure  of  independence.     For  a 
given  body  of  documents,  a  statistical  measure  of  co-occurrence 
can  be  derived  that  can  be  compared  mth  the  measure  of  inde- 
pendence.    Each  pair  of  tenns  A  and  B  can  be  said  to  be  more 
or  less  associated  depending  on  the  excess  of  their  statistical 
measure  over  the  independence  value.     The  measure  of  association 
is  conputed  for  all  pairs  of  terms  for  sn  entire  collection. 
Then,   for  a  given  search,  the  number  of  documents  can  be  ex- 
panded oy  ejgjanding  the  query  to  include  terras  liiglily  asso- 
ciated Tdth  those  that  malce  up  the  initial  query.     Tiiis 
expansion  by  means  of  associative  searching  can  be  approached 
using  a  chain  strategy,  wherein  the  search  is  carided  out  for 
the  original  term,  then  the  tenn  most  higlily  associated  idth 
the  original  term,  then  the  term  most  higlily  associated  i-dth 
the  first  associated  term,   and  so  on,   or  a  hub  strategy,  irherein 
the  search  is  eujpanded  first  xath  the  term  most  higlily  correlated 
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Tdoh  the  original  ter%  then  the  second  moet  higlily  associated 
tonu  Tjith  the  original  terra,   aiid  so  on,   or  by  some  combination 
of  these  txio  strategies.     ICulins  cleiras  that  all  "reasonable" 
measvii-'es  o£  association  are  of  the  forras 

Excess  over  independence  value  of  A  and  B 

A  fimction  of  the  four  catcgoi-les  u,v,::,   and  y, 

uhere:     ::  =  A  and  B'^  y  =  not  A  and  not  Bj  u  =  A  and  not  Bj 
V  =  not  A  and  B. 

Cooper  (I971-2itl)  proposes  a  measure  of  association  in  document 
space  obtained  by  computing  the  degree  of  association  betireen 
pairs  of  documents  for  the  classes  of  documents  inde:ced  by 
t\io  different  inde::  terrtis.     One  of  the  measures  of  association 
proposed  by  Kulins  can  then  be  applied  to  get  a  document  to 
docuiiient  correlation  for  all  pairs  of  documents  in  the  collection. 

Maron  and  Kuhns   (I960)  propose  a  technique  called  "probabilistic 
inde:d.ng"  wherein  a  comparative  concept  of  relevance  is 
developed  based  on  theory  of  pi-obability  and  used  for  indexing 
and  searcliing  in  an  autom.ated  document  retrieval  system.     In 
this  teclinique,   the  computer  computes  a  "relevance"  nmnber 
for  each  document  in  the  collection  in  response  to  s  given 
request,   and  then  ranlcs  the  entire  collection  by  probable 
relevance.     This  approach  is  one  of  statistical  closeness  rather 
than  semantic  closenessj  it  interprets  a  request  as   a  clue 
on  which  to  base  statistical  inference. 

Closeness  and  the  concepts  of  associative  search  and  probab- 
ilistic indexing  are  attempts  to  deal  i-.dth  the  problem  of  re- 
levance.    A  system  should  access  those  documents  in  response 
to  a  given  request  that  are  relevantj  i.e.,  it  should  sttenpt 
to  maximize  the  number  of  dociiments  that  are  relevant-and- 
retrieved  and  mininiLze  the  number  of  documents  that  are  not- 
relevant-and-retrieved  and  that  are  relevant-and-not-retrieved. 
Common  measures  of  the  effectiveness  of  a  system  in  retrieval 
are  those  of  precision  (avoidance  of  retrieving  irrelevant 
documents)   and  recall   (avoidance  of  not  retrieving  relevant 
documents).     Cooper  (1968)  proposes  a  measure  of  retrievel 
effectiveness  based  on  expected  search  length,   based  on  the 
e:q)ected  niimber  of  irrelevant  documents  in  a  collection  that 
would  have  to  be  searched  through  before  a  specified  desired 
number  of  relevant  documents  coiild  be  found.     The  advantages 
of  Cooper's  "expected  search  length  reduction  factor"  over 
the  traditional  measures  ere  in  providing  a  single  measure  of 
performance,     gradations  of  retrieval  status,   evaluation  re- 
lative to  random  searching,   and  the  taldng  into  account  of 
the  amount  of  relevant  material  desired  by  the  user.     Cooper 
(1970)   goes  further  to  explore  the  relations  between  expected 
search  length  as  a  performance  measure  and  system  chsractei'lstics 
of  three  tj^pes  of  retrieval  systems:      a  single-term- request. 
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direct  match  system,   a  multiple-term-request,   simple  overlap 
system,   and  xrfiat  he  describes  as  a  "quasi-Boolean"  overlap 
system  that  allows  the  prefixing  of  the  one  or  more  terms  in 
the  request  by  "not".     These  case  studies  uere  undertaken  to 
explore  the  effect  of  various  system  characteristics  on 
retrieval  effectiveness, 

Relevance  can  be  viewed  as  being  either  logical  in  nature, 
based  on  the  topic  itself,   and  relations  of  "appropriateness", 
"pertinence",  or  "direct  answer-ness"  between  the  query  and  a 
document I   or  utilitarian,   based  on  recency,  quality,   accuracy, 
language  in  wliich  written,   and  authorship.     Maron  and  Kuhns 
(i960)  in  their  paper  on  Probabilistic  Indexing  develop  a 
probability-based  comparative  concept  of  relevance  that 
incorporates  statistical  measures  of  user  satisfaction  xd-th 
documents  in  a  collection.     Relevance  in  this  view  is  roughly 
the  probability  of  satisfjdng  an  inforraation  need.     Cooper 
(1969)   describes  a  precise  definition  of  relevance  based  on 
his  logical  consequence  detecting  question  answering  sj'^stem 
(I96I1)  5     that  is,   that  a  stored  statement  is  relevant  to  a 
query  if  and  only  if  it  is  a  member  of  a  minimal  premise  set 
of  which  the  query  is  a  logical  consequence  for  either  the 
query  or  its  negation.     This  is  a  mathematically  precise  de- 
finition only  for  certain  inferential  data  retrieval  systems, 
but  can  be  conceptually  extended  to  reference  retrieval  systems: 
"A  docuraent  is  relevant  to  an  information  need  if  and  only 
if  it  contains  at  least  one  sentence  wliich  is  relevant  to 
that  need."      (Cooper,  I969,  p.  10).     Cooper  concludes  by 
contrasting  utility  and  relevance  in  measuring  the  performance 
of  retrieval  systems  and  states  that  perhaps  the  relevance  of 
"relevance"  is  that  it  is  the  most  important  factor  in  utility 
(1969,  pp.  21-22).     An  important  factor  in  the  utility  of  a 
specific  document  to  a  specific  request  might  be  its  dependency 
on  previously  read  documents!   document  D2  iTiight  be  useful 
only  after  Dl  has  been  read,  or  might  be  useful  only  if  Dl 
has  not  been  read,   and  so  on.     Kuhns  describes  a  measure  of 
performance  related  to  the  xd-llingness  of  the  user  to  process 
more  than  a  certain  number  of  documents  before  reaching  a 
minim-um  number  of  relevant  documents  (unpubl.  M3A-TRW  report j 
Maron,  1971-2 U2). 

Useful  measures  of  retrieval  effectiveness  should  be  comparative, 
should  be  quantitative,   and  perhaps  should  carry  some  concept 
of  the  cost  of  using  the  system.     Swanson  (I96O)  incorporates 
a  penalty  for  irrelevance  into  the  m.easure  of  effectiveness. 
Sxjets   (1963,  1969)  places  emphasis  on  the  need  for  the  measure 
of  performance  to  be  a  single  niunber,   rather  than  a  set  of 
numbers  such  as  precision  and  recall  measures.     He  describes 
a  probabilistic  approach  to  optimizing  system  performance. 
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Salton  (1969)  compares  conventional,   manual  document  indexing 
by  hiMnsii  indexers  mth  fully  automatic  text  processing,   and 
concludes  that  fully  automatic  text  analysis  and  seaTch  systems 
do  not  appear  to  perform  in  a  manner  inferior  to  manual  docu- 
ment indexing  and  search  formulation,   and  that  vrhile  manual 
systems  can  range  from  very  excellent  to  very  poor,   depending 
on  operator  awareness  of  user  needs  and  the  stored  collection, 
automated  systems  perform  more  or  less  consistently.     Automatic 
indexing  can  be  approached  by  simply  throi.dng  out  all  connectives 
and  non-content  xrords  and  listing  the  remaining  substance 
words,   and  counting  the  number  of  times  that  they  occur.     The 
result  would  be  a  list  of  index  terms  ranked  by  their  frequency 
of  occurrence  in  the  document.     Given  search  queries  ;ri.th 
enough  clues   (terms)   and  such  modifications  as  weighted  index 
terms,   an  automated  indexing  and  searcliing  system  should  be 
able  to  do  very  well  in  corrparison  to  manual  indexing  and 
searching  systems.     Kuhn  distinguishes  between  derivative 
indexing  (using  terms  derived  from  the  text)   and  assignment 
indexing  (using  terms  from  a  standard  list) (1^62),    and 
formulates  the  problem  as  being  to  disregard  syntactic  function 
words  and  infrequently  occurring  xrords  so  that  the  remaining 
high  frequency  xjords  right  give  a  reasonable  picture  of 
"aboutness".     /imong  several  objections  to  this  definition 
(Stevens,  1970,  p.  173)  is  the  one  that  the  most  information 
might  xjell  be  conveyed  by  the  least  frequently  occurring 
terms.     Stevens  describes  a  large  quantity  of  research  into 
the  area  of  automated  indexing,   and  concludes  that  the  field 
is  already  at  the  stage  x\rhere  usable  applications  can  be  made, 
and  foresees  expansion  of  the  Held  in  order  to  gain  the 
benefits  of  more  detailed  and  broader  indexing  xd.th  given 
resources  and  easy  updating  and  revision  of  indexing. 

I.J.  Good  cites  several  fields  x\rherein  the  concept  of  clustering 
is  being  explored.     A  cluster  is  simply  a  group  of  similar  in- 
dividuals, x-rliich  in  information  retrieval  might  be  either 
index  terras  or  documents.     SearcMng  in  a  clustered  collection 
would  be  in  cluster  sets.     Libraries  have  traditionally  been 
divided  into  subjectsj  perhaps  it  x-roxold  be  beneficial  to  make 
the  division  in  terms  of  similar  (clustered)  vectors  of 
descriptors.     Clustering  for  information  retrieval  vrould  be 
superfluous  if  infinite  search  speeds  were  available,   but 
since  they  are  not,  it  may  prove  to  be  a  useful  technique. 
Clustering  techniques  are  described  by  Ball   (I966),   Sokal 
(1966),  Bonner  (196U)   and  MacNaughton-Smith  (1965). 

Another  approach  to  searching  ndght  be  formulated  based  on 
the  user's  making  inferences  about  the  documents  that  he  thinks 
x«all  satisfy  his  information  need,   based  on  qualities  that 
might  accompany  the  desired  traits.     In  tliis  approach,   the 
user  might  put  together  a  series  of  contextual  characteristics, 
such  as  "people  xirho  have  co-authored  X'.dth....y   "publish  in 
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the  journal  of...."j,   aiid  so  on.     To  this  end  e  relational 
language,  or  calciilus  of  relations  irdght  prove  useful. 

Cooper  (1969,  p.  Ill)  characterises  the  difference 

betueen  question  answering  (inferential)  systems  and  literature 
searcliing  (reference  retrieval)   systems  as  folloxjss     "In  the 
inferential  sj'^sten,  the  only  information  used  in  deducing 
an  answer  is  information  stored  xdthin  the  system  memory, 
and  all  of  the  logical  deductions  are  also  carried  out  by 
the  system  itself.     In  a  reference  retrieval  system,  the  user's 
ovm  information  in  his  personal  memory  is  also  brought  into 
play,  and  the  user  contributes  his  oxm  powers  of  reasoning 
to  the  deductive  process  as  well.     Thus  the  stored  information 
available  for  satisfjdng  an  information  need  xjith  the  help 
of  a  refex-ence  retrieval  system  must  be  viewed  as  the  user's 
ox'jn  Icnoxiledge  plus  the  system  stored  data,   and  the  deductive 
apparatus  is  the  user's  oxm  deductive  power  plus  x^^hatever 
deductive  poxjer  has  been  programmed  into  the  system." 

Hote  that  the  manner  in  x-jhich  Cooper  defines  the  use  of  a 
reference  retrieval  system  closely  parallels  characterizations 
of  information  in  the  planning  process:     one's  values  or 
x-JorldviexTj   combined  x-dth  factual  data,  to  yield  plans  for 
action  in  bringing  about  a  desired  situation.     One  might 
conclude  from  tMs,   that  the  greatest  usefxiLness  of  formal 
information  retrieval  systems  in  planning  and  design  x-dll  be 
in  the  use  of  reference  retrieval  systems  to  provide  factual 
data  for  interpretation  by  the  user's  deontic  (ought-to-be) 
knoxTledgej   and  that  the  uses  of  question  answering  systems 
xdll  probably  be  on  only  the  simpler  levels,  parallel  to 
airline  reservation  systems,  in  xjliich  the  emphasis  is  on 
putting  standard  queries  to  constantly  changing  data  bases. 

Ilaron  (1971-21+2)  describes  the  field  of  information  retrieval 
as  being  on  a  plateau  at  present.     A  large  quantity  of  xjork 
has  been  done  over  the  last  10-1^  years,  but  noxj  there  seems 
to  be  a  lack  of  progress.     Perhaps  xdiat  is  lacldng  is  a 
theoretical  basis  for  the  next  brealcthroughs,   and  that  basis 
must  come  in  some  area  such  as  the  study  of  information  trajisfer 
in  the  brain.     One  area  of  great  interest  in  the  near  futxire 
in  the  field  of  information  retrieval  is  likely  to  be  that 
of  undesirable  side  effects  such  as  authoritarian  r.dsuse  of 
data  on  individuals,   accessible  and  manipulable  in  a  xjay  that 
has  never  before  been  the  case.     There  is  cu.rrent  interest 
in  the  technological  forecasting  of  such  abuses  and  hox'j-  they 
might  be  controlled. 
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